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Abstract. Let G be a connected, simply connected one-parameter metabelian nilpotent 
Lie group, that means, the corresponding Lie algebra has a one-codimensional abelian sub- 
algebra. In this article we show that G contains a discrete cocompact subgroup. Given 
a discrete cocompact subgroup F of G, we define the quasi-regular representation t = Indp 1 
of G. The basic problem considered in this paper concerns the decomposition of r into 
irreducibles. Wc give an orbital description of the spectrum, the multiplicity function and 
we construct an explicit intertwining operator between t and its desintegration without 
considering multiplicities. Finally, unlike the Moore inductive algorithm for multiplicities 
on nilmanifolds, we carry out here a direct computation to get the multiplicity formula. 
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1 Introduction 

Let G be a connected simply connected nilpotent Lie group having a cocompact discrete sub- 
group r, then the nilmanifold G/T has a unique invariant Borel probability measure v and G 
acts on L^(G/r) = L^(G/r, v) by the quasi-regular representation r = Indp 1. This means that 

[r{a)f){g) = fia-'g), f G L\G/T), a,geG. 

It is known [8l p. 23] that the representation r splits into a discrete direct sum of a countable 
number of irreducible unitary representations, each vr with finite multiplicity m(7r). We write 



7rG(G:r) 



m(7r)7r. 



(1) 



For any given nilpotent Lie group G with discrete cocompact subgroup F, there are two general 
problems to consider. The first is to determine the spectrum {G : F) and the multiplicity 
function m(7r) of these representations. The second is to construct an explicit intertwining 
operator between r and its decomposition into irreducibles. 

A necessary and sufficient condition for vr to occur in r was obtained, by Calvin C. Moore |14j . 
in the special case in which F is a lattice subgroup of G (i.e., F is a discrete cocompact subgroup 
of G and log(F) is an additive subgroup of the Lie algebra g of G). Moreover, Moore deter- 
mined an algorithm which expresses m(7r) in terms of multiplicities of certain representations of 
a subgroup of codimension one in G. Later, R. Howe and L. Richardson I18| independently 
obtained a closed formula for the multiplicities for general F. In [T5], C. Moore and J. Wolf 
determine explicitly which square integrable representations of G occur in the decomposition ([T]) 
and give a method for calculating the multiplicities. Using the Poisson summation and Selberg 
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trace formulas, L. Corwin and F. Greenleaf gave a formula for m(7r) that depended only on the 
coadjoint orbit in g* corresponding to vr via Kirillov theory and the structure of T [B] (see also 



Our main goals in this paper are twofold. The first is to give an orbital description of 
the decomposition of r into irreducibles, in the case when G is a connected, simply connected 
one-parameter metabelian nilpotent Lie group. Such a decomposition has two components, 
the spectrum (G : T) and the multiplicity function vr i— > m(7r). We give orbital description 
of both. The second main goal is to give an explicit intertwining operator between r and its 
desintegration. 

This paper is organized as follows. In Section [2l we establish notations and recall a few stan- 
dard facts about representation theory, rational structure and cocompact subgroups of a con- 
nected simply connected nilpotent Lie groups. Section [3] is devoted to present some results 
which will be used in the next sections. In Section |4] we prove, first, that a one-parameter 
metabelian nilpotent Lie group admits a discrete uniform subgroup F (i.e., the homogeneous 
space G/T is compact). Next, for a fixed discrete uniform subgroup F we prove the existence of 
a one-codimensional abelian rational ideal M (i.e., M n F is a discrete uniform subgroup of M). 
Furthermore, we give a necessary and sufficient condition for the uniqueness of M. In Section[5l 
we pick from a strong Malcev basis strongly based on F, an orbital description of the spectrum 
(G : F). We obtain the following decomposition 



where S is a crosssection for F-orbits in a certain subspace W C g* and V C g* (more details in 
Theorem H]). 

We describe an intertwining operator U of r and p, defined for all ^ € C{G/T) and g ^ G hy 



where M{1) = Af if / € S or M{1) = G if Z € V. This operator does not take into account the 
multiplicities of the decomposition of r. As a consequence, we give an orbital description of the 
multiplicity function. 

2 Notations and basic facts 

The purpose of this section is to establish notations which will be used in the sequel and recall 
some basic definitions and results which we shall freely use afterwards without mentioning them 
explicitly. 

Let G be a connected and simply connected nilpotent Lie group with Lie algebra g, then the 
exponential map exp : g — ?> G is a diffeomorphism. Let log : G ^ g denote the inverse of exp. 

2.1 Induced representation 

Starting from a closed subgroup H oi a, nilpotent Lie group G and a unitary representation a 
of i^ in a Hilbert space M'a, let us construct a unitary representation of G. We realize the 
unitary representation Ind^ a of G, the representation induced on G from the representation a 
of by left translations on the completion of the space of continuous functions F on G with 
values in J^a- satisfying 



laiaEiEiE]). 



p = 01nd^ Xz + 0Xi, 





F{gh) = a{h-'){F{g)) 



g eG, h£ H, 



(2) 
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and having a compact support modulo H, provided with the norm 

lli^ll = K^dlFin)^ = ( / \\Fig)fduG,H{9) 

\Jg/h 

where 

((lndga)(5)F)(x) = F(g-ix), g,xeG. 

2.2 The orbit theory 

Suppose G is a nilpotent Lie group with Lie algebra q. G acts on q (respectively g*) by the 
adjoint (respectively co-adjoint) action. For / € g*, let 

5(/) = {Xe5: {l,[X,Q]) = {0}} 
be the stabilizer of / in g which is actually the Lie algebra of the Lie subgroup 

Gil) = {geG: g-l = l}. 
So, it is clear that g(/) is the radical of the skew-symmetric bilinear form Bi on g defined by 

Bi{X,Y) = {l,[X,Y]), X,YeQ. 

A subspace b{l) of the Lie algebra g is called a polarization for Z G g* if it is a maximal 
dimensional isotropic subalgebra with respect to Bi. We have the following equality 

dim(b(/)) = i(dim(0)-|-dim(g(O)). (3) 

If b{l) is a polarization for I G g* let B{1) = exp(b(/)) be the connected subgroup of G with 
Lie algebra b{l) and define a character of B{1) by the formula 

X/(exp(X)) = e2-<W, VXGb(/). 

Now, we recall the Kirillov orbital parameters. We denote by G the unitary dual of G, i.e. 
the set of all equivalence classes of irreducible unitary representations of G. We shall sometimes 
identify the equivalence class [vr] with its representative vr and we denote the equivalence relation 
between two representations vri and 7r2 by vri ~ 7r2 or even by vri = ^2- The dual space G of G 
is parameterized canonically by the orbital space Q*/G. More precisely, for Z € g* we may find 
a real polarization b{l) for /. Then the representation tti = Ind^^^-j xi is irreducible; its class is 
independent of the choice of b{l); the Kirillov mapping 

Kirc : g* ^ G, I ^ ni 

is surjective and factors to a bijection g*/G — )■ G. Given vr S G, we write il(7r) € g*/G to denote 
the inverse image of tt under the Kirillov mapping Kirc. 

2.3 Rational structures and uniform subgroups 

In this section we present some results on discrete uniform subgroups of nilpotent Lie groups. 

Let G be a nilpotent, connected and simply connected real Lie group and let g be its Lie 
algebra. We say that g (or G) has a rational structure if there is a Lie algebra gq over Q such 
that g = gQ <8> H. It is clear that g has a rational structure if and only if g has an R-basis 
{Xi, . . . , Xn) with rational structure constants. 

A discrete subgroup T is called uniform in G if the quotient space G/T is compact. The 
homogeneous space G/T is called a compact nilmanifold. A proof of the next result can be 
found in Theorem 7 of |12] or in Theorem 2.12 of [17J. 
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Theorem 1 (the Malcev rationaUty criterion). Let G be a simply connected nilpotent Lie group, 
and let g be its Lie algebra. Then G admits a uniform subgroup T if and only if q admits a basis 
{Xi, . . . , Xn) such that 

n 

[Xi.Xj] = ^ CijkXk V 1 < z, j < n, 
fc=i 

where the constants Cijk are all rational. 

More precisely, if G has a uniform subgroup F, then q has a rational structure such that 

0Q = 0Q,r = Q-span{log(r)} . 

Conversely, if q has a rational structure given by some Q-algebra C g, then G has a uniform 
subgroup r such that log(r) C Qq (see [2] or [T2]). 

Definition 1 ([2]). Let g be a nilpotent Lie algebra and let ^ = {Xi, . . . be a basis of g. 
We say that ^ is a strong Malcev basis for g if g^ = ]R-span{Xi, . . . ,Xi} is an ideal of g for 
each 1 < i < n. 

Let r be a discrete uniform subgroup of G. A strong Malcev basis (Xi, . . . for g is said 
to be strongly based on T if 

r = exp(ZXi) • • • exp(ZX„). 

Such a basis always exists (see [2| [T3]). 

2.4 Rational subgroups 

Definition 2 (rational subgroup). Let G be a connected simply connected nilpotent Lie group 
with Lie algebra g. We suppose that g has rational structure given by qq. 

(1) Let f) be an R-subspace of g. We say that t) is rational if f) = Il-span{f)Q} where l)q = 
f) ngq. 

(2) A connected, closed subgroup H oi G is called rational if its Lie algebra f) is rational. 

Remark 1. The R-span and the intersection of rational subspaces are rational [21 Lemma 5.1.2]. 

Definition 3 (subgroup with good P-heredity, [E]). Let F be a discrete uniform subgroup in 
a locally compact group G and H a closed subgroup in G. We say that H is a subgroup with 
good F-heredity if the intersection T D H is a discrete uniform subgroup of H. 

Theorem 2 ([H Lemma A. 5]). Let G be a connected, simply connected nilpotent Lie group 
with Lie algebra q, let T be a discrete uniform subgroup of G, and give g the rational structure 
0Q = Q-span {log(r)}. Let H be a Lie subgroup of G. Then the following statements are 
equivalent 

(1) H is rational; 

(2) H is a subgroup with good T-heredity; 

(3) The group H is V -closed {i.e., the set HT is closed in G). 

A proof of the next result can be found in Proposition 5.3.2 of [2]. 
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Proposition 1. Let T he discrete uniform subgroup in a nilpotent Lie group G, and let Lii ^ 
H2 ^ ••• ^ -f^fc = G be rational normal subgroups of G. Let f)i, . . . , f)fc_i, fj^ = q be the 
corresponding Lie algebras. Then there exists a strong Malcev basis {Xi, . . . , X„) for q strongly 
based on T and passing through . . . , tjk-i- 

Now let G be a connected, simply connected nilpotent Lie group with Lie algebra g, and 
suppose that q has a rational structure given by the discrete uniform subgroup F. A real linear 
functional / € 3* is called rational (/ € g^, gq = Q-span {log(r)}) if (/, g^) C Q, or equivalently 

(/,iog(r)) cQ. 

Proposition 2 ([6], Theorem A. 7). Let G be a nilpotent Lie group with rational structure and 
let g be its Lie algebra. If I Q* is rational, then its radical q{1) is rational. 

3 Preliminary results 

Definition 4. A functional / S g* is said in general position or generic linear functional, if its 
coadjoint orbit has maximum dimension. 

The following proposition will be used in the sequel 

Proposition 3. Let G = exp(g) be a nilpotent Lie group and T a discrete uniform subgroup 
of G. Then there exist rational generic linear functionals. 

Proof. Let O be the set of elements in general position in g*. We have O is a nonempty Zariski 
open set in g*. Since g^ is dense in g* then g^ n O 7^ 0. ■ 

Before stating the next result, we introduce some notations and definitions. We first recall 
the concept of fundamental domain. 

Definition 5. Let G be a topological group and let if be a subgroup of G. A fundamental 
domain for is a Borel subset Vt oi G such that 

h&H 

is the disjoint union of the Borel subsets Vth^ h ^ H. 

Remark 2. It is clear that a Borel subset of G is a fundamental domain for if in G if and 
only if the natural map — ?> G/H defined by 1-^ gH is bijective. 

Let r be a discrete uniform subgroup of a connected, simply connected nilpotent Lie group G, 
and let ^ = {Xi, . . . ,Xn) be a strong Malcev basis for the Lie algebra g of G strongly based 
on r. Define the mapping : R" G hy 

E,0s{T) = exp(i„X„) • • • exp(iiXi), 
where T = (ii, . . . , t„) € R". It is well known that is a diffeomorphism [2]. Let 

I = [0, 1) = G R : 0<t<l} 
and let 



n = E^(r). 
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Then J7 is a fundamental domain for F in G Lemma 3.6], and the mapping E,^ maps the 
Lebesgue measure dt on I" to the G-invariant probabihty measure on G/T, that is, for ip 
in C{G/T), we have 



ip{g)di^ig) = / ^(E^it))dt. 

G/T JI" 
Furthermore, for (f) € Cc{G) we have 

/ cP{g)du{g) = V / 4>{EMt)B,^{s))dt. (4) 
Jg Ji- 

The following proposition will be used in the sequel. 

Proposition 4. Let G = exp(g) be a connected, simply connected nilpotent Lie group and 
let T be a discrete uniform subgroup of G. Let m be a rational ideal of q of dimension k. Let 
M = exp(m) and let {Xi, . . . , Xn) be a strong Malcev basis of q strongly based on F passing 
through m. For every ^ G Cc{G/M), we have 



/ i{g)dg = / U exp(x„_fcX„)---exp(xiXfc+i) 

J G/M sez--^-^^"-' ^ 

X exp{sn-kXn) ■ • •exp(siXfc+i) • • • dxk+i, 



where Xi are the image of Xi under the canonical projection p : — t- s/m. 

Proof. Since M is rational, it follows from Lemma 5.1.4 of that P{T) is a uniform subgroup 
of P{G), where P : G ^ G/M is the canonical projection. On the other hand, {X^+i, ■ ■ ■ ,Xn) 
is a strong Malcev basis of g/m strongly based on P{T). We conclude by applying ■ 



4 On the rational structure of one- parameter 
metabelian nilpotent Lie groups 

Definition 6 (one-parameter metabelian nilpotent Lie algebra). A nonabelian nilpotent Lie al- 
gebra Q is said to be one parameter metabelian nilpotent Lie algebra, if it admits a co-dimensional 
one abelian ideal in g. 

First, we give an important example of one-parameter metabelian nilpotent Lie groups. 

Example 1 (the generic filiform nilpotent Lie group). Let G be the generic filiform nilpotent 
Lie group of dimension n with Lie algebra g, where 

g = ]R-span{Xi, . . . 

The Lie brackets given by 

[Xn,Xi] = Xi_i, i = 2,...,n-l, 

and the nondefined brackets being equal to zero or obtained by antisymmetry. It is clear that g 
is a one-parameter metabelian nilpotent Lie algebra. 

Remark 3. Let g be a one-parameter metabelian nilpotent Lie algebra. Any codimension one 
abelian ideal m C g is a common polarization for all functionals / € g* in general position (i.e.. 
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Definition 7 (one-parameter metabelian nilmanifold). A factor space of a one-parameter 
metabelian nilpotent Lie group over a discrete uniform subgroup is called a one-parameter 
metabelian nilmanifold. 

Proposition 5. Let q he a one-parameter metabelian nilpotent Lie algebra. Then we have the 
following decomposition 

such that for all i = 1, . . . ,p, the subalgebra M^XoCm is the generic filiform nilpotent Lie algebra 
of dimension Ui + 1 and a C 3(0)- 

Proof. Let 3 be a one-codimensional abelian ideal of q and let X € 3 such that 

= J © R-span {X} . 
Let adXjj be the restriction of adX to 3; 

adX\~j: 3^3, Y ^ [X,Y]. 

Remark that adX acts as a nilpotent linear transformation on 3. By the Jordan normal form 
theorem, the matrix of adXj^ is similar to a matrix in real Jordan canonical form. Then there 
exist = (ei, . . . , e^-i) a basis of 3, Jo,ni, 1 < * < elementary Jordan blocks of order nj such 
that 

Mat(adX|f,^3) = diag[Jo,„i, . . . , Jo,nJ- 
This completes the proof. ■ 

Proposition 6. Let G be a one-parameter metabelian nilpotent Lie group with Lie algebra Q. 
Then G admits a discrete uniform subgroup. 

Proof. This follows at once from the Malcev rationality criterion (Theorem [T|) and Proposi- 
tion El ■ 

Proposition 7. Let G be a one-parameter metabelian nilpotent Lie group of dimension n. Let T 
be a discrete uniform subgroup of G. Then G admits a rational abelian ideal of codimension one. 

Before proving the proposition, we need the following theorem. 

Theorem 3 (pj p. 17]). If I Q* is in general position then g(/) is abelian. 

Proof of Proposition O Let (ei, . . . , e^) be a strong Malcev basis for q strongly based on F. 
Let / be a rational linear functional in general position (see Proposition [3|) . The stabilizer of / 
is a rational abelian subalgebra. On the other hand, applying equality ([3|) we obtain 

dim(0(l)) =n-2. (5) 

Let (cj^, 643) be a basis of g modulo g(/). If [e^^, g(/)] = {0}, then g(/) © R-spanjej-^ } is a rational 
abelian ideal of g of codimension one. If [eij,g(/)] / {0}, let 

a={X e E-span{eii,eiJ : [X,q{1)] = {0}}. 
In this case, we have dim(a) = 1. Moreover, it is clear that 

a={X eg: [X, g(/)] = {0}} n K-span{ei, , e^J. 
By Proposition 5 of |9], the space 

{XGg: [X,g(/)] = {0}} 

is rational. Consequently, a is rational subspace in g. Thus a © g(^) is a rational abelian ideal 
of g of codimension one. ■ 
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The next proposition is a generalization of Proposition 3.1 of [10], in which we give a necessary 
and sufficient condition for uniqueness of the one-codimensional abehan normal subgroup. 

Proposition 8. Let G be a one-parameter metabelian nilpotent Lie group. Then G admits 
a unique one-codimensional abelian normal subgroup if and only if G is not of the form x R'^, 
where fc G IN and H3 is the Heisenberg group of dimension 3. 

Proof. The necessity of this condition is evident. We prove the sufficiency. Let mi, m2 be 
two abelian one-codimensional subalgebras of g. Let I be a linear functional in general position. 
Since every polarization for / contains then there exists X G g (see ([5|)) such that 

mi = 0(0 ffi lR.-span {X} . 
Let y € such that 

= mi © R-span {Y} . 
On the other hand, there exist a G R* and u € mi such that 

iTi2 = 0(0 ® R-span {aY + u} . 
Since mi and m2 are abelian then 



Consequently, [X,y] is nonzero bracket. Let a be an ideal of such that 0(Z) = a©R-span 

Then we have = f)3 ffi a where f)3 = R-span {X, y, [X, Y\} is the three dimensional Heisenberg 



5 Construction of intertwining operators 

Let G be a one-parameter metabelian nilpotent Lie group of dimension n with Lie algebra g. 
Let r be a discrete uniform subgroup of G. Let M = exp(m), where m is a rational abelian ideal 
in of codimension one. Let = {Xi, . . . ,Xn) be a strong Malcev basis of strongly based 
on r passing through [g, g] and m. We put 



[Y,m = {^}- 



algebra. 



[g, g] = R-span{Xi, . . . , Xp] 



(6) 



and 



m 



R-span {Xi, . . . ,X„„i} . 



Let 



V 



'LXl 




and 



In the following, for Z € m* C g* and g G we denote 



Ad; 



*ogl = {Ad* gl)\ 



Lemma 1. The subset W is AdgL invariant. 
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Proof. Let exp(7) G F and / G W. Let i = l,...,n — 1. By definition of the coadjoint 
representation we have 

(exp(-7)./,X,) = (/,e^'i^(X,)). 
On the other hand, since M is normal in G, we have that 

exp (e^^T(X)) = exp(7) exp Xi exp(-7) G T n M. 
Since M is abelian and 

r n M = exp(ZXi) • • • exp{ZXn-i) 

then 

log(r) n m = zxi e • • • e zXn-i. 

It follows that (/,e^'i^(Xi)) G Then 

exp(-7) •/ G ZXi*©---eZX*_i. (7) 
It remains to prove that 

(exp(-7)./)|[g,g]/0. (8) 
We have 

5(exp(-7) • /) = Adexp(-7)(0(/)). 

Since g{l) / g, then g(exp(— 7) • /) / g and therefore ([S]) holds. Consequently, from ([7]) and ([5]) 
we have exp(— 7) • I G W. ■ 

Let C{G/T) be the space of all complex valued continuous functions ^ on G satisfying 

^(57) = ag), (9) 

for any g in G and 7 in F. 
Lemma 2. Let / G V, we /laue 

(1) X/|r = l. 

(2) For G C{G/T), and g G, the function 

G/r^C, mV ^ ^{gm)xi{m) 
is well defined. 

Proof. (1) Let 7 G F, then there exist ti, . . . , S ^ such that 

7 = exp(iiXi) • • • exp(t„X„). 
By the Baker-Campbell-Hausdorff formula we have 

log(7) = tp+iXp+i H h tnXn (mod [g, g]). 

Since /|[g^g] = then xiil) = 1- 
(2) Let 7 G F. We have 

i{gm'y)xi{m-i) = (,{gm)xi{m-y) ^^=® i{gm)xi{m)xi{l) = i{gm)xi{m). 
This completes the proof of the lemma. ■ 
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Let S be a crosssection for T-orbits in W. Let 



We are now in a position to formulate the following 
Theorem 4. The operator U defined for all ^ € C{G/T) and g ^ G hy 



u(6(0(5) = <^ 



^{gm)xi{m)drh ifleT., 
M/Mnr 

i{gm)xi{m)drn ifleV 
G/T 



is an isometric linear operator having value in the Hilbert space J^p of p and can be extended 
on L^(G/r) to an intertwining operator of t and p. 

Proof. Clearly for / € S U V, the function U(^)(/) satisfies the covariance relation First, 
we establish that U is well defined and isometric. Let ^ G C{G/T). Then 

II u(6f = E II u(e)(/)iii.(G/M,o + E II • 

«gs lev 
We proceed to calculate the first sum. For x — (^i? • • • ? ^n—i 

) e E"~^ and t G E, let 
5(t, x) = E,^((x, t)) = exp(tX„) exp(x„„iX„„i) • • • exp(xiXi), 



Eiiu(e)(oiii2 



(G/M,0 



E 
E 



\mmi9)\'dg 



M/Mnr 



^{9m)xi{m)dm 



dg 



M/Mf\r 



i{5{t,Q)5{s,0)m)xi{m)dm 



dt 



l&T. sew. 

Y.Y. 

ZGE sGZ 



M/MnV 



^{6{t, 0)m)xi{S{s, 0)-^m6{s, 0))(im 



dt 



M/Mnr 



^{S{t, 0)m)xAd*(5(s,o))«("i)c^"i 



dt. 



As the mapping S x Z — > W, i-^ Ad*((5(s, 0))/ is bijective, then 



Eiiu(e)(Oiii.(G/M,o = E 

«gs /gw 

= E 

lew 



M/Mnr 



^{6{t, 0)m)xi{rn)dm 



dt 



aSit,x))xiiSiO,x))dx 



dt. 



Next, we compute 

Eiiu(e)(Of = Eiu(e)(0(e)p = E / ^(9)xi{9)dg 

]^^'> i^^i I JG/r 



lev 



lev 



lev 
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E 



^{6{t,x))xiiSit,x))dtdx 



I" 



^{^{t, x))xh (^(0, x))xi^ (<5(t, 0))dtdx 



= E E 

(where Vi = 'LX*^^ + ■■■ + V2 = 'EX*^ and l = h+l2(^Vi® V2) 



E 



e((^(t,x))x/iWO,x))dx 



dt, 



where we have apphed Parseval's equahty with respect to the variable t. Summarizing, we have 

2 



+ E 



^{of = y,l I mt,x))ximx))da 



dt 



mt,^))xi{mx))dx 



dt. 



It is clear that W and Vi are disjoint. In fact, suppose that WnVi 7^ 0. Let / G WnVi. 
The condition / G Vi implies that l\[g^g] = since [g,g] = RXi © ••• ® RXp (see (0)). This 
contradicts the fact that ^|[g,g] / because / G W. Now, let 



= W U Vi . 
It is clear that 

m*^ = ZXl + ... + ZX:__,. 
Consequently, we obtain 



(10) 



u(OII 



E 



e(5(t,x))xK'J(0,:r))dx 



dt 



|e(5(t,x))pdx(it 



(by Parseval's equality) 



\mt,x))\'dxdt = m%^^. 



The operator U being now isometric, it can be extended to an isometry of L^(G/r). It remains to 
verify that U is an intertwining operator for r and p. It suffices then to prove that U oT{g){S,) = 
p{g) o U(^) for every g in G and ^ in C{G/T). Let / G S, a G G. We compute 



Uor(<7)(0(0(«) 



'T{g){i){am)xi{m)dm = / ^{g ^am)xi{m)dm. 

M/Ainr JM/Mnv 



On the other hand 

p(5) oU(0(/)(a) = lnd%xi{9){^m)){a) = U(O(0(5-'a) 

^ {g~^am)xi {m)drh. 



M/A/nr 



Thus 



Uor(5)(0(0(«)=P(5)oU(e)(/)(a). 
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Similarly, we prove that the same equality holds if / G V. Consequently, the following diagram: 



U 



U 



(11) 



is commutative. ■ 
Next, we show that U is an invertible operator. For this, let 

i& lev 

where Cc{G/M, xi) is the space of complex valued continuous functions ^ on G satisfying 

CW = Xr'Me(5) V<7GG, meM, (12) 
and having a compact support modulo M. 
Lemma 3. Let K € M'^p, I € S and g ^ G. The function 

r/r n M ^ c, 7(r n m) ^ k{1){97) 

is well defined. 

Proof. Let 7 G r and 7' G T n M. Since K{1) G CciG/M,xi) then K{1) satisfies the covari- 
ance relation (I12p . Hence K{l){g^j') = xi^{l')K{l){g^). On the other hand, as L n M = 
exp(ZXi) • • • exp(ZX„_i) and M is abehan, then log(r) n m = ZXi ® • • • © It follows 

that X/|rnA/ = 1, in particular, xiW) = 1- Then K{l){gjy) = K{l){g-f). ■ 

Let 

J^l = {K G : K{iyis a zero function everywhere, except for finite number of 
The space J^^p is dense in p. 
Lemma 4. For K G J^^p and g €z G, the sum 

^ Km^) + Y,Kmg), geG, (13) 
ies 7er/rnM /ev 

has only finitely many nonzero terms. 
Proof. Let / G S U V. For g £ G, let 
Sg = {jer: 57 G supp(K(0)} . 

Then there is an integer n^(;) independent of g (z G such that Sg is the union of at most n^^^) 
cosets of r n M [31 Lemma 3.2]. This observation shows that the sum over T/T f] M in ()13p 
has at most n^^^-j nonzero entries for each g £ G. Consequently, the sum (jlSp is a sum of finite 
terms. ■ 



We define the operator V on the dense subspace Jf^ by 

YiK)ig)=Y^ E Kil)igj) + ^Kil)ig), g e G. 
«gs 7Gr/rnM lev 
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Proposition 9. The operator V is the inverse of JJ. 

Proof. First, we observe that V(i(') satisfies the covariance relation ([2]) in L^(G/r). In fact, 
let g G and 70 € F, we have 

V(JO(57o) = Y1 E ^(0(5707) + ^Kil)ig^o) 
ies 7er/rnM lev 

(in the first sum, we use the change of variable 7 1— ?> ^q^j) 

ies 7Gr/rnM zgv 
As 7o S F and I G V then Xiilo) = 1 (see property (1) of Lemma[2]). Consequently we obtain 
Y{K){gJo)=^r{K){g). 

Now, we calculate UoV(i^) for some K G Jifp. Let K G °. We distinguish the following 
two cases. 

Case 1. Iq G S. 



UoV(K)(/o)(e) = U(V(i^))(/o)(e) = / Y{K){m)xi,{m)dm 

JM/AlnT 

= 1 (EE m{rni) + Y.K{l){ni)]xi,{ra)dm 

E E ^(0(77"^"i7)+^K(0(m) JxioMdm 

i^(/)(7)xr'(7-^n7) + 5]i^(/)(e)xr'M)xJm 
. ies 7Gr/rnM /gv / 

/ (EE ^'(0(7)XAd*7K"i) + Vi^(/)(e)x«('^) IXio("^)'^^ 

iM/Afnr V ^gj. ^er/rnM «gv / 

E E -^(0(7)/ XAd*7K"^)XioM'^"^ 

«6E7gr/rnAf ^A//A/nr 

+ E^(^)(^) / Xl{m)xiQ{m)dm. 
JM/MnT 



A//A/nr 



A//A/nr 



'A//A/nr 

The interchange of integration and summation is justified by the fact that K G J^^- On the 
other hand, the integral 

1, if Ad* 7/ = lo, 
0, otherwise. 



XAd* -,lim)xio{'m)dm 

M/MnT 



Since I and Iq belong to S, then 

Ad* 7/ = /o ^ = ^0 and 7(F n M) = F n M. 
It follows that 

E E ^(0(7)/ XAd*7/Mx«o("^)f^"^ = ^(^o)(e). 

^T^/T.^.^ JM/Mnr 



iGS 7Gr/rnAf 
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On the other hand, since Iq then 

JM/Mnr 
for every / S V and hence 

Y]iC(/)(e) / xi{m)xio{.m)dm = 0. 

JM/Mnr 

Finally, we obtain 

VoW{K){h){e)=K{h){e). 

Case 2. Iq G V. 

Uo V(A')(Zo)(e) = U(V(i^))(/o)(e) = / \ {K){g)xi,{g)d9 

JG/r 

First remark that: 



^K{l)ig)xi,ig)dg= [ J]K(0(e)xi(5)x«o(<?)^5 
G/r Jc/r 

^i^(0(e) / MxMdg = K{lo){e). 
1~r, J G/r 



lev 

Next 



■^G/r 7Gr/rnM zes 7Gr/rnA/ 



= I K{l){6{t,0h^-'6{0,xh)xiom,0))xkM0,x))dxdt 
•^^ «GS7gr/rnM 

= /E E K{i){6{t,o)^) [ XAd'Amx))xioimx))dxxiomo))dt = o. 

■^izeS7er/rnM -^i"'' 

Then Uo V(K)(/o)(e) = i^(/o)(e). We conclude that for every in we have Uo V(K) = K. 
Next, for ^ G L2(G/r) such that U(C) G we compute 

VoU(e)(e)=V(U(e))(e) = J] U(e)(0(7) + E U(O(0(e) 

iGE 7er/rnA/ lev 

= E E / e(7Hx/(m)dm + j;U(e)(0(e) 

= E E / e(7m7-')XiM'^m + J]U(O(0(e) 

(GE 7Gr/rnA'/ •^A//A-/nr 
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= E E / eMx/(7"'rn7)dm + ^U(e)(/)(e) 

= T. I ar^^)XAd'^l{m)dm + Y,'^mi){e) 

«GE 7er/rnM WA-^nr 

= E / ?('^(0,x))x/(<5(0,x))dx + ^U(O(0(e). 

^^^^ /TTn— 1 

On the other hand 



= J2Y. mt,x))xh{mx))xi,mo))dtdx 

(where Vi = + • • • + ZX*_i,V2 = ^X* and / = /i + G Vi V2) 



V / e(<5(0,x))xz,(<5(0,x))dx 

V / e(<5(0,x))xK'^(0,x))d:c. 



by the Fourier inversion 
formula to the variable t 



Then 



VoU(e)(e) = V / e(<5(0,x))xK'^(0,^))d:r + V / e(<5(0,x))xK5(0,x))(ia 

by the Fourier inversion 
bv mOI — T f , , , , , , , , formula to the variable x , , 

= E / e(5(0,x))xz(<5(0,x))dx = e(e). 



FinaUy, we obtain that U is well defined and isometric and has dense range. It therefore 
extends uniquely into an isometry of M'^ onto ^ p. As immediate consequences of the last 
theorem are the following 

Corollary 1. We have the following decomposition 

T~p = 0Indf^X/ + 0Xi- 

/es lev 

For the next corollary, we write i^E to denote the cardinal number of a set E. 

Corollary 2. We keep the same notations as Theorem^ The multiplicity function t<i[tti) is 
given by m(7rz) = 1 ifleV and m{Tri) = #[r2(7r/) n S] ifleT,. 

Proof. Let / G A, where A = S or V. The multiplicity ^(Tr/) is equal to the number oil' ^ A 
such that TT; ~ Till. Thus, by the Kirillov theory, we have 

m(7r;) = #{/' G A : r G ^{1^1)] = n A]. 

Now, if / G V, the coadjoint orbit ^{tti) has only one element and therefore T<i{'Ki) = 1. ■ 



16 



A. Ghorbel 



References 

[1] Bernat P., Conze N.. Duflo M., Levy-Nahas M., Rais M., Renouard P., Vergne M., Representations des 
groupes de Lie resolubles, Monographies de la Societe Mathematique de France, no. 4, Dunod, Paris, 1972. 

[2] Corwin L., Greenleaf P.P., Representations of nilpotent Lie groups and their applications. Part 1. Basic 
theory and examples, Cambridge Studies in Advanced Mathematics, Vol. 18, Cambridge University Press, 
Cambridge, 1990. 

[3] Corwin L., Greenleaf P.P., Intertwining operators for representations induced from uniform subgroups, 
\Acta Math.M m (1976), 275-301. 

[4] Corwin L., Greenleaf P.P., Integral formulas with distribution kernels for irreducible projections in of 
a nilmanifold. lJ. Fund. Anan 23 (1976), 255-284. 

[5] Corwin L., Greenleaf P.P., Penney R., A canonical formula for the distribution kernels of primary projections 
in of a nilmanifold, I Comm. Pure Appl. M ath. 30 (1977), 355-372. 

[6] Corwin L., Greenleaf P.P., Character formulas and spectra of compact nilmanifolds, IJ. Fund. Anal. \ 21 
(1976), 123-154. 

[7] Corwin L., Greenleaf P.P., Penney R., A general character formula for irreducible projections on of 
a nilmanifold. Math. Ann. 225 (1977), 21-32. 

[8] Gel'fand I.M., Graev M.L, Piatetskii-Shapiro I.L, Representation theory and automorphic functions, 
W.B. Saunders Co., Philadelphia, Pa. - London - Toronto, Ont. 1969. 

[9] Ghorbel A., Hamrouni H., Discrete cocompact subgroups of the five-dimensional connected and simply 
connected nilpotent Lie groups. .SIGMA 5 (2009), 020, 17 pages, arXiv:0902.2 977 

[10] Hamrouni H., Discrete cocompact subgroups of the generic filiform nilpotent Lie groups, J. Lie Theory 18 
(2008), 1-16. 

[11] Howe R., On Frobenius reciprocity for unipotent algebraic group over Q. \Amer. J. Maf/J 93 (1971), 163-172. 

[12] Malcev A.L, On a class of homogeneous spaces, Amer. Math. Soc. Transl. (1951), no. 39, 33 pages. 

[13] Matsushima Y., On the discrete subgroups and homogeneous spaces of nilpotent Lie groups, Nagoya Math. J. 
2 (1951), 95-110. 

[14] Moore C.C., Decomposition of unitary representations defined by discrete subgroups of nilpotent groups, 
\Ann. of Math~( 2) 82 (1965), 146-182. 

[15] Moore C.C., Wolf J., Square integrable representations of nilpotent groups. [Trans. Amer. Math. Soc.\ 185 
(1973), 445-462. 

[16] Onishchik A.L., Vinberg E.B., Lie groups and Lie algebras. 11. Discrete subgroups of Lie groups and coho- 
mologies of Lie groups and Lie algebras. Encyclopaedia of Mathematical Sciences, Vol. 21, Springer- Verlag, 
Berhn, 2000. 

[17] Raghunathan M.S., Discrete subgroups of Lie groups, Ergebnisse der Mathematik und ihrer Grenzgebiete, 
Band 68, Springer- Verlag, New York - Heidelberg, 1972. 

[18] Richardson L., Decomposition of the I/^-space of a general compact nilmanifolds, I Amer. J. Matfe.1 93 (1971), 
173-190. 



